Web Data Management Web Search

For personal use only, not for distribution. 6 Further Reading 35 With a constantly increasing size of dozens of billions of freely accessible documents, one of the major issues raised by the World Wide Web is that of searching in an effective and efficient way through these documents to find those that best suit a user's need. The purpose of this chapter is to describe the techniques that are at the core of today's search engines (such as Google 1 , Yahoo! 2 , Bing 3 , or Exalead 4), that is, mostly keyword search in very large collections of text documents. We also briefly touch upon other techniques and research issues that may be of importance in next-generation search engines. This chapter is organized as follows. In Section 1, we briefly recall the Web and the languages and protocols it relies upon. Most of these topics have already been covered earlier in the book, and their introduction here is mostly intended to make the present chapter self-contained. We then present in Section 2 the techniques that can be used to retrieve pages from the Web, that is, to crawl it, and to extract text tokens from them. First-generation search engines, exemplified by Altavista 5 , mostly relied on the classical information retrieval (IR) techniques, applied to text documents, that are described in Section 3. The advent of the Web, and more generally the steady growth of documents collections managed by institutions of all kinds, has led to extensions of these techniques. We address scalability issues in Section 3.3, with focus on centralized indexing. Distributed approaches are investigated in Chapter ??. The graph structure of the Web gives rises to ranking techniques that very 3 effectively complement information retrieval. We conclude with a brief discussion of currently active research topics about Web search in Section 5. 1 The World Wide Web Whereas the Internet is a physical network of computers (or hosts) connected to each other from all around the world, the World Wide Web, WWW or Web in short, is a logical collection of hyperlinked documents shared by the hosts of this network. A hyperlinked document is just a document with references to other documents of the same collection. Note that documents of the Web may refer both to static documents stored on the hard drive of some host of the Internet and to dynamic documents that are generated …